Global Model Interpretation via Recursive Partitioning

In this work, we propose a simple but e‚ective method to interpret black-box machine learning models globally. Œat is, we use a compact binary tree, the interpretation tree, to explicitly represent the most important decision rules that are implicitly contained in the black-box machine learning models. Œis tree is learned from the contribution matrix which consists of the contributions of input variables to predicted scores for each single prediction. To generate the interpretation tree, a uni€ed process recursively partitions the input variable space by maximizing the di‚erence in the average contribution of the split variable between the divided spaces. We demonstrate the e‚ectiveness of our method in diagnosing machine learning models on multiple tasks. Also, it is useful for new knowledge discovery as such insights are not easily identi€able when only looking at single predictions. In general, our work makes it easier and more ecient for human beings to understand machine learning models.